Skip lead_time=0h for ECMWF max/min step_type variables#492
Merged
Conversation
…n job 10fg (wind gust) is a max-window variable absent from the GRIB index at lead_time=0h since there is no prior post-processing step at initialization time. The operational update was generating source file coords for all lead times including 0h, causing a KeyError when looking up '10fg' in the index. Fix by adding has_hour_0_values() to ecmwf_utils following the same pattern as noaa_utils, and filtering lead_time=0h coords in generate_source_file_coords for any variable group where not all variables have hour 0 values. Update the slow integration test to use a dynamic recent init_time (instead of hardcoded 2026-01-01) and test via generate_source_file_coords over lead_times [0h, 3h] so it would catch this class of bug in the future. https://claude.ai/code/session_01LBZoe7ccmmgZxdKD8uHhZr
- source_groups now groups by (date_available, has_hour_0_values) so all variables in a group share the same hour-0 behaviour, allowing generate_source_file_coords to use item() for a clean single-value check - Add tests/ecmwf/test_ecmwf_utils.py with parametrized unit tests for has_hour_0_values covering all step_types and the override mechanism https://claude.ai/code/session_01LBZoe7ccmmgZxdKD8uHhZr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes handling of ECMWF variables with
step_type="max"orstep_type="min"by skippinglead_time=0hduring source file coordinate generation, since these variables have no value at initialization time.Key Changes
New utility function
has_hour_0_values()inecmwf_utils.pyto determine if a variable has a value atlead_time=0hFalsefor max/min step_type variables (no prior post-processing step at t=0)Truefor avg/accum variables (include 0h accumulation in GRIB)hour_0_values_overridein variable internal attributesUpdated region job to skip
lead_time=0hwhen any variable in a group lacks hour 0 valuesEnhanced integration test to validate variables across multiple lead times
lead_time=0handlead_time=3hto catch format changesImplementation Details
The fix ensures that when a data variable group contains any max/min step_type variable, the entire group skips
lead_time=0hduring coordinate generation, preventing downstream errors when attempting to read unavailable data from ECMWF files.https://claude.ai/code/session_01LBZoe7ccmmgZxdKD8uHhZr